Textbook Practice

The unnest_tokens function

# load text
text <- c("Because I could not stop for Death -", "He kindly stopped for me -", "The Carriage held but just Ourselves -",
    "and Immortality")
text
## [1] "Because I could not stop for Death -"  
## [2] "He kindly stopped for me -"            
## [3] "The Carriage held but just Ourselves -"
## [4] "and Immortality"
# put text into data frame
library(dplyr)
text_df <- tibble(line = 1:4, text = text)
text_df
line text
1 Because I could not stop for Death -
2 He kindly stopped for me -
3 The Carriage held but just Ourselves -
4 and Immortality
# tokenize & transform into a tidy data structure - unnest_tokens() function
library(tidytext)
text_df %>%
    unnest_tokens(word, text)
line word
1 because
1 i
1 could
1 not
1 stop
1 for
1 death
2 he
2 kindly
2 stopped
2 for
2 me
3 the
3 carriage
3 held
3 but
3 just
3 ourselves
4 and
4 immortality

Tidying the works of Jane Austen

library(janeaustenr)
library(dplyr)
library(stringr)

# add columns for row and chapter numbers for lines of text
original_books <- austen_books() %>%
    group_by(book) %>%
    mutate(linenumber = row_number(), chapter = cumsum(str_detect(text, regex("^chapter [\\divxlc]",
        ignore_case = TRUE)))) %>%
    ungroup()

original_books
text book linenumber chapter
SENSE AND SENSIBILITY Sense & Sensibility 1 0
Sense & Sensibility 2 0
by Jane Austen Sense & Sensibility 3 0
Sense & Sensibility 4 0
Sense & Sensibility 5 0
Sense & Sensibility 6 0
Sense & Sensibility 7 0
Sense & Sensibility 8 0
Sense & Sensibility 9 0
CHAPTER 1 Sense & Sensibility 10 1
Sense & Sensibility 11 1
Sense & Sensibility 12 1
The family of Dashwood had long been settled in Sussex. Their estate Sense & Sensibility 13 1
was large, and their residence was at Norland Park, in the centre of Sense & Sensibility 14 1
their property, where, for many generations, they had lived in so Sense & Sensibility 15 1
respectable a manner as to engage the general good opinion of their Sense & Sensibility 16 1
surrounding acquaintance. The late owner of this estate was a single Sense & Sensibility 17 1
man, who lived to a very advanced age, and who for many years of his Sense & Sensibility 18 1
life, had a constant companion and housekeeper in his sister. But her Sense & Sensibility 19 1
death, which happened ten years before his own, produced a great Sense & Sensibility 20 1
alteration in his home; for to supply her loss, he invited and received Sense & Sensibility 21 1
into his house the family of his nephew Mr. Henry Dashwood, the legal Sense & Sensibility 22 1
inheritor of the Norland estate, and the person to whom he intended to Sense & Sensibility 23 1
bequeath it. In the society of his nephew and niece, and their Sense & Sensibility 24 1
children, the old Gentleman’s days were comfortably spent. His Sense & Sensibility 25 1
attachment to them all increased. The constant attention of Mr. and Sense & Sensibility 26 1
Mrs. Henry Dashwood to his wishes, which proceeded not merely from Sense & Sensibility 27 1
interest, but from goodness of heart, gave him every degree of solid Sense & Sensibility 28 1
comfort which his age could receive; and the cheerfulness of the Sense & Sensibility 29 1
children added a relish to his existence. Sense & Sensibility 30 1
Sense & Sensibility 31 1
By a former marriage, Mr. Henry Dashwood had one son: by his present Sense & Sensibility 32 1
lady, three daughters. The son, a steady respectable young man, was Sense & Sensibility 33 1
amply provided for by the fortune of his mother, which had been large, Sense & Sensibility 34 1
and half of which devolved on him on his coming of age. By his own Sense & Sensibility 35 1
marriage, likewise, which happened soon afterwards, he added to his Sense & Sensibility 36 1
wealth. To him therefore the succession to the Norland estate was not Sense & Sensibility 37 1
so really important as to his sisters; for their fortune, independent Sense & Sensibility 38 1
of what might arise to them from their father’s inheriting that Sense & Sensibility 39 1
property, could be but small. Their mother had nothing, and their Sense & Sensibility 40 1
father only seven thousand pounds in his own disposal; for the Sense & Sensibility 41 1
remaining moiety of his first wife’s fortune was also secured to her Sense & Sensibility 42 1
child, and he had only a life-interest in it. Sense & Sensibility 43 1
Sense & Sensibility 44 1
The old gentleman died: his will was read, and like almost every other Sense & Sensibility 45 1
will, gave as much disappointment as pleasure. He was neither so Sense & Sensibility 46 1
unjust, nor so ungrateful, as to leave his estate from his nephew;–but Sense & Sensibility 47 1
he left it to him on such terms as destroyed half the value of the Sense & Sensibility 48 1
bequest. Mr. Dashwood had wished for it more for the sake of his wife Sense & Sensibility 49 1
and daughters than for himself or his son;–but to his son, and his Sense & Sensibility 50 1
son’s son, a child of four years old, it was secured, in such a way, as Sense & Sensibility 51 1
to leave to himself no power of providing for those who were most dear Sense & Sensibility 52 1
to him, and who most needed a provision by any charge on the estate, or Sense & Sensibility 53 1
by any sale of its valuable woods. The whole was tied up for the Sense & Sensibility 54 1
benefit of this child, who, in occasional visits with his father and Sense & Sensibility 55 1
mother at Norland, had so far gained on the affections of his uncle, by Sense & Sensibility 56 1
such attractions as are by no means unusual in children of two or three Sense & Sensibility 57 1
years old; an imperfect articulation, an earnest desire of having his Sense & Sensibility 58 1
own way, many cunning tricks, and a great deal of noise, as to outweigh Sense & Sensibility 59 1
all the value of all the attention which, for years, he had received Sense & Sensibility 60 1
from his niece and her daughters. He meant not to be unkind, however, Sense & Sensibility 61 1
and, as a mark of his affection for the three girls, he left them a Sense & Sensibility 62 1
thousand pounds a-piece. Sense & Sensibility 63 1
Sense & Sensibility 64 1
Mr. Dashwood’s disappointment was, at first, severe; but his temper was Sense & Sensibility 65 1
cheerful and sanguine; and he might reasonably hope to live many years, Sense & Sensibility 66 1
and by living economically, lay by a considerable sum from the produce Sense & Sensibility 67 1
of an estate already large, and capable of almost immediate Sense & Sensibility 68 1
improvement. But the fortune, which had been so tardy in coming, was Sense & Sensibility 69 1
his only one twelvemonth. He survived his uncle no longer; and ten Sense & Sensibility 70 1
thousand pounds, including the late legacies, was all that remained for Sense & Sensibility 71 1
his widow and daughters. Sense & Sensibility 72 1
Sense & Sensibility 73 1
His son was sent for as soon as his danger was known, and to him Mr.  Sense & Sensibility 74 1
Dashwood recommended, with all the strength and urgency which illness Sense & Sensibility 75 1
could command, the interest of his mother-in-law and sisters. Sense & Sensibility 76 1
Sense & Sensibility 77 1
Mr. John Dashwood had not the strong feelings of the rest of the Sense & Sensibility 78 1
family; but he was affected by a recommendation of such a nature at Sense & Sensibility 79 1
such a time, and he promised to do every thing in his power to make Sense & Sensibility 80 1
them comfortable. His father was rendered easy by such an assurance, Sense & Sensibility 81 1
and Mr. John Dashwood had then leisure to consider how much there might Sense & Sensibility 82 1
prudently be in his power to do for them. Sense & Sensibility 83 1
Sense & Sensibility 84 1
He was not an ill-disposed young man, unless to be rather cold hearted Sense & Sensibility 85 1
and rather selfish is to be ill-disposed: but he was, in general, well Sense & Sensibility 86 1
respected; for he conducted himself with propriety in the discharge of Sense & Sensibility 87 1
his ordinary duties. Had he married a more amiable woman, he might Sense & Sensibility 88 1
have been made still more respectable than he was:–he might even have Sense & Sensibility 89 1
been made amiable himself; for he was very young when he married, and Sense & Sensibility 90 1
very fond of his wife. But Mrs. John Dashwood was a strong caricature Sense & Sensibility 91 1
of himself;–more narrow-minded and selfish. Sense & Sensibility 92 1
Sense & Sensibility 93 1
When he gave his promise to his father, he meditated within himself to Sense & Sensibility 94 1
increase the fortunes of his sisters by the present of a thousand Sense & Sensibility 95 1
pounds a-piece. He then really thought himself equal to it. The Sense & Sensibility 96 1
prospect of four thousand a-year, in addition to his present income, Sense & Sensibility 97 1
besides the remaining half of his own mother’s fortune, warmed his Sense & Sensibility 98 1
heart, and made him feel capable of generosity.– “Yes, he would give Sense & Sensibility 99 1
them three thousand pounds: it would be liberal and handsome! It would Sense & Sensibility 100 1
be enough to make them completely easy. Three thousand pounds! he Sense & Sensibility 101 1
could spare so considerable a sum with little inconvenience.”– He Sense & Sensibility 102 1
thought of it all day long, and for many days successively, and he did Sense & Sensibility 103 1
not repent. Sense & Sensibility 104 1
Sense & Sensibility 105 1
No sooner was his father’s funeral over, than Mrs. John Dashwood, Sense & Sensibility 106 1
without sending any notice of her intention to her mother-in-law, Sense & Sensibility 107 1
arrived with her child and their attendants. No one could dispute her Sense & Sensibility 108 1
right to come; the house was her husband’s from the moment of his Sense & Sensibility 109 1
father’s decease; but the indelicacy of her conduct was so much the Sense & Sensibility 110 1
greater, and to a woman in Mrs. Dashwood’s situation, with only common Sense & Sensibility 111 1
feelings, must have been highly unpleasing;–but in HER mind there was Sense & Sensibility 112 1
a sense of honor so keen, a generosity so romantic, that any offence of Sense & Sensibility 113 1
the kind, by whomsoever given or received, was to her a source of Sense & Sensibility 114 1
immovable disgust. Mrs. John Dashwood had never been a favourite with Sense & Sensibility 115 1
any of her husband’s family; but she had had no opportunity, till the Sense & Sensibility 116 1
present, of shewing them with how little attention to the comfort of Sense & Sensibility 117 1
other people she could act when occasion required it. Sense & Sensibility 118 1
Sense & Sensibility 119 1
So acutely did Mrs. Dashwood feel this ungracious behaviour, and so Sense & Sensibility 120 1
earnestly did she despise her daughter-in-law for it, that, on the Sense & Sensibility 121 1
arrival of the latter, she would have quitted the house for ever, had Sense & Sensibility 122 1
not the entreaty of her eldest girl induced her first to reflect on the Sense & Sensibility 123 1
propriety of going, and her own tender love for all her three children Sense & Sensibility 124 1
determined her afterwards to stay, and for their sakes avoid a breach Sense & Sensibility 125 1
with their brother. Sense & Sensibility 126 1
Sense & Sensibility 127 1
Elinor, this eldest daughter, whose advice was so effectual, possessed Sense & Sensibility 128 1
a strength of understanding, and coolness of judgment, which qualified Sense & Sensibility 129 1
her, though only nineteen, to be the counsellor of her mother, and Sense & Sensibility 130 1
enabled her frequently to counteract, to the advantage of them all, Sense & Sensibility 131 1
that eagerness of mind in Mrs. Dashwood which must generally have led Sense & Sensibility 132 1
to imprudence. She had an excellent heart;–her disposition was Sense & Sensibility 133 1
affectionate, and her feelings were strong; but she knew how to govern Sense & Sensibility 134 1
them: it was a knowledge which her mother had yet to learn; and which Sense & Sensibility 135 1
one of her sisters had resolved never to be taught. Sense & Sensibility 136 1
Sense & Sensibility 137 1
Marianne’s abilities were, in many respects, quite equal to Elinor’s. Sense & Sensibility 138 1
She was sensible and clever; but eager in everything: her sorrows, her Sense & Sensibility 139 1
joys, could have no moderation. She was generous, amiable, Sense & Sensibility 140 1
interesting: she was everything but prudent. The resemblance between Sense & Sensibility 141 1
her and her mother was strikingly great. Sense & Sensibility 142 1
Sense & Sensibility 143 1
Elinor saw, with concern, the excess of her sister’s sensibility; but Sense & Sensibility 144 1
by Mrs. Dashwood it was valued and cherished. They encouraged each Sense & Sensibility 145 1
other now in the violence of their affliction. The agony of grief Sense & Sensibility 146 1
which overpowered them at first, was voluntarily renewed, was sought Sense & Sensibility 147 1
for, was created again and again. They gave themselves up wholly to Sense & Sensibility 148 1
their sorrow, seeking increase of wretchedness in every reflection that Sense & Sensibility 149 1
could afford it, and resolved against ever admitting consolation in Sense & Sensibility 150 1
future. Elinor, too, was deeply afflicted; but still she could Sense & Sensibility 151 1
struggle, she could exert herself. She could consult with her brother, Sense & Sensibility 152 1
could receive her sister-in-law on her arrival, and treat her with Sense & Sensibility 153 1
proper attention; and could strive to rouse her mother to similar Sense & Sensibility 154 1
exertion, and encourage her to similar forbearance. Sense & Sensibility 155 1
Sense & Sensibility 156 1
Margaret, the other sister, was a good-humored, well-disposed girl; but Sense & Sensibility 157 1
as she had already imbibed a good deal of Marianne’s romance, without Sense & Sensibility 158 1
having much of her sense, she did not, at thirteen, bid fair to equal Sense & Sensibility 159 1
her sisters at a more advanced period of life. Sense & Sensibility 160 1
Sense & Sensibility 161 1
Sense & Sensibility 162 1
Sense & Sensibility 163 1
CHAPTER 2 Sense & Sensibility 164 2
Sense & Sensibility 165 2
Sense & Sensibility 166 2
Mrs. John Dashwood now installed herself mistress of Norland; and her Sense & Sensibility 167 2
mother and sisters-in-law were degraded to the condition of visitors. Sense & Sensibility 168 2
As such, however, they were treated by her with quiet civility; and by Sense & Sensibility 169 2
her husband with as much kindness as he could feel towards anybody Sense & Sensibility 170 2
beyond himself, his wife, and their child. He really pressed them, Sense & Sensibility 171 2
with some earnestness, to consider Norland as their home; and, as no Sense & Sensibility 172 2
plan appeared so eligible to Mrs. Dashwood as remaining there till she Sense & Sensibility 173 2
could accommodate herself with a house in the neighbourhood, his Sense & Sensibility 174 2
invitation was accepted. Sense & Sensibility 175 2
Sense & Sensibility 176 2
A continuance in a place where everything reminded her of former Sense & Sensibility 177 2
delight, was exactly what suited her mind. In seasons of cheerfulness, Sense & Sensibility 178 2
no temper could be more cheerful than hers, or possess, in a greater Sense & Sensibility 179 2
degree, that sanguine expectation of happiness which is happiness Sense & Sensibility 180 2
itself. But in sorrow she must be equally carried away by her fancy, Sense & Sensibility 181 2
and as far beyond consolation as in pleasure she was beyond alloy. Sense & Sensibility 182 2
Sense & Sensibility 183 2
Mrs. John Dashwood did not at all approve of what her husband intended Sense & Sensibility 184 2
to do for his sisters. To take three thousand pounds from the fortune Sense & Sensibility 185 2
of their dear little boy would be impoverishing him to the most Sense & Sensibility 186 2
dreadful degree. She begged him to think again on the subject. How Sense & Sensibility 187 2
could he answer it to himself to rob his child, and his only child too, Sense & Sensibility 188 2
of so large a sum? And what possible claim could the Miss Dashwoods, Sense & Sensibility 189 2
who were related to him only by half blood, which she considered as no Sense & Sensibility 190 2
relationship at all, have on his generosity to so large an amount. It Sense & Sensibility 191 2
was very well known that no affection was ever supposed to exist Sense & Sensibility 192 2
between the children of any man by different marriages; and why was he Sense & Sensibility 193 2
to ruin himself, and their poor little Harry, by giving away all his Sense & Sensibility 194 2
money to his half sisters? Sense & Sensibility 195 2
Sense & Sensibility 196 2
“It was my father’s last request to me,” replied her husband, “that I Sense & Sensibility 197 2
should assist his widow and daughters.” Sense & Sensibility 198 2
Sense & Sensibility 199 2
“He did not know what he was talking of, I dare say; ten to one but he Sense & Sensibility 200 2
# restructure to one-token-per-row - in this case one word per row
library(tidytext)
tidy_books <- original_books %>%
    unnest_tokens(word, text)

tidy_books
book linenumber chapter word
Sense & Sensibility 1 0 sense
Sense & Sensibility 1 0 and
Sense & Sensibility 1 0 sensibility
Sense & Sensibility 3 0 by
Sense & Sensibility 3 0 jane
Sense & Sensibility 3 0 austen
Sense & Sensibility 5 0 1811
Sense & Sensibility 10 1 chapter
Sense & Sensibility 10 1 1
Sense & Sensibility 13 1 the
Sense & Sensibility 13 1 family
Sense & Sensibility 13 1 of
Sense & Sensibility 13 1 dashwood
Sense & Sensibility 13 1 had
Sense & Sensibility 13 1 long
Sense & Sensibility 13 1 been
Sense & Sensibility 13 1 settled
Sense & Sensibility 13 1 in
Sense & Sensibility 13 1 sussex
Sense & Sensibility 13 1 their
Sense & Sensibility 13 1 estate
Sense & Sensibility 14 1 was
Sense & Sensibility 14 1 large
Sense & Sensibility 14 1 and
Sense & Sensibility 14 1 their
Sense & Sensibility 14 1 residence
Sense & Sensibility 14 1 was
Sense & Sensibility 14 1 at
Sense & Sensibility 14 1 norland
Sense & Sensibility 14 1 park
Sense & Sensibility 14 1 in
Sense & Sensibility 14 1 the
Sense & Sensibility 14 1 centre
Sense & Sensibility 14 1 of
Sense & Sensibility 15 1 their
Sense & Sensibility 15 1 property
Sense & Sensibility 15 1 where
Sense & Sensibility 15 1 for
Sense & Sensibility 15 1 many
Sense & Sensibility 15 1 generations
Sense & Sensibility 15 1 they
Sense & Sensibility 15 1 had
Sense & Sensibility 15 1 lived
Sense & Sensibility 15 1 in
Sense & Sensibility 15 1 so
Sense & Sensibility 16 1 respectable
Sense & Sensibility 16 1 a
Sense & Sensibility 16 1 manner
Sense & Sensibility 16 1 as
Sense & Sensibility 16 1 to
Sense & Sensibility 16 1 engage
Sense & Sensibility 16 1 the
Sense & Sensibility 16 1 general
Sense & Sensibility 16 1 good
Sense & Sensibility 16 1 opinion
Sense & Sensibility 16 1 of
Sense & Sensibility 16 1 their
Sense & Sensibility 17 1 surrounding
Sense & Sensibility 17 1 acquaintance
Sense & Sensibility 17 1 the
Sense & Sensibility 17 1 late
Sense & Sensibility 17 1 owner
Sense & Sensibility 17 1 of
Sense & Sensibility 17 1 this
Sense & Sensibility 17 1 estate
Sense & Sensibility 17 1 was
Sense & Sensibility 17 1 a
Sense & Sensibility 17 1 single
Sense & Sensibility 18 1 man
Sense & Sensibility 18 1 who
Sense & Sensibility 18 1 lived
Sense & Sensibility 18 1 to
Sense & Sensibility 18 1 a
Sense & Sensibility 18 1 very
Sense & Sensibility 18 1 advanced
Sense & Sensibility 18 1 age
Sense & Sensibility 18 1 and
Sense & Sensibility 18 1 who
Sense & Sensibility 18 1 for
Sense & Sensibility 18 1 many
Sense & Sensibility 18 1 years
Sense & Sensibility 18 1 of
Sense & Sensibility 18 1 his
Sense & Sensibility 19 1 life
Sense & Sensibility 19 1 had
Sense & Sensibility 19 1 a
Sense & Sensibility 19 1 constant
Sense & Sensibility 19 1 companion
Sense & Sensibility 19 1 and
Sense & Sensibility 19 1 housekeeper
Sense & Sensibility 19 1 in
Sense & Sensibility 19 1 his
Sense & Sensibility 19 1 sister
Sense & Sensibility 19 1 but
Sense & Sensibility 19 1 her
Sense & Sensibility 20 1 death
Sense & Sensibility 20 1 which
Sense & Sensibility 20 1 happened
Sense & Sensibility 20 1 ten
Sense & Sensibility 20 1 years
Sense & Sensibility 20 1 before
Sense & Sensibility 20 1 his
Sense & Sensibility 20 1 own
Sense & Sensibility 20 1 produced
Sense & Sensibility 20 1 a
Sense & Sensibility 20 1 great
Sense & Sensibility 21 1 alteration
Sense & Sensibility 21 1 in
Sense & Sensibility 21 1 his
Sense & Sensibility 21 1 home
Sense & Sensibility 21 1 for
Sense & Sensibility 21 1 to
Sense & Sensibility 21 1 supply
Sense & Sensibility 21 1 her
Sense & Sensibility 21 1 loss
Sense & Sensibility 21 1 he
Sense & Sensibility 21 1 invited
Sense & Sensibility 21 1 and
Sense & Sensibility 21 1 received
Sense & Sensibility 22 1 into
Sense & Sensibility 22 1 his
Sense & Sensibility 22 1 house
Sense & Sensibility 22 1 the
Sense & Sensibility 22 1 family
Sense & Sensibility 22 1 of
Sense & Sensibility 22 1 his
Sense & Sensibility 22 1 nephew
Sense & Sensibility 22 1 mr
Sense & Sensibility 22 1 henry
Sense & Sensibility 22 1 dashwood
Sense & Sensibility 22 1 the
Sense & Sensibility 22 1 legal
Sense & Sensibility 23 1 inheritor
Sense & Sensibility 23 1 of
Sense & Sensibility 23 1 the
Sense & Sensibility 23 1 norland
Sense & Sensibility 23 1 estate
Sense & Sensibility 23 1 and
Sense & Sensibility 23 1 the
Sense & Sensibility 23 1 person
Sense & Sensibility 23 1 to
Sense & Sensibility 23 1 whom
Sense & Sensibility 23 1 he
Sense & Sensibility 23 1 intended
Sense & Sensibility 23 1 to
Sense & Sensibility 24 1 bequeath
Sense & Sensibility 24 1 it
Sense & Sensibility 24 1 in
Sense & Sensibility 24 1 the
Sense & Sensibility 24 1 society
Sense & Sensibility 24 1 of
Sense & Sensibility 24 1 his
Sense & Sensibility 24 1 nephew
Sense & Sensibility 24 1 and
Sense & Sensibility 24 1 niece
Sense & Sensibility 24 1 and
Sense & Sensibility 24 1 their
Sense & Sensibility 25 1 children
Sense & Sensibility 25 1 the
Sense & Sensibility 25 1 old
Sense & Sensibility 25 1 gentleman’s
Sense & Sensibility 25 1 days
Sense & Sensibility 25 1 were
Sense & Sensibility 25 1 comfortably
Sense & Sensibility 25 1 spent
Sense & Sensibility 25 1 his
Sense & Sensibility 26 1 attachment
Sense & Sensibility 26 1 to
Sense & Sensibility 26 1 them
Sense & Sensibility 26 1 all
Sense & Sensibility 26 1 increased
Sense & Sensibility 26 1 the
Sense & Sensibility 26 1 constant
Sense & Sensibility 26 1 attention
Sense & Sensibility 26 1 of
Sense & Sensibility 26 1 mr
Sense & Sensibility 26 1 and
Sense & Sensibility 27 1 mrs
Sense & Sensibility 27 1 henry
Sense & Sensibility 27 1 dashwood
Sense & Sensibility 27 1 to
Sense & Sensibility 27 1 his
Sense & Sensibility 27 1 wishes
Sense & Sensibility 27 1 which
Sense & Sensibility 27 1 proceeded
Sense & Sensibility 27 1 not
Sense & Sensibility 27 1 merely
Sense & Sensibility 27 1 from
Sense & Sensibility 28 1 interest
Sense & Sensibility 28 1 but
Sense & Sensibility 28 1 from
Sense & Sensibility 28 1 goodness
Sense & Sensibility 28 1 of
Sense & Sensibility 28 1 heart
Sense & Sensibility 28 1 gave
Sense & Sensibility 28 1 him
Sense & Sensibility 28 1 every
Sense & Sensibility 28 1 degree
Sense & Sensibility 28 1 of
Sense & Sensibility 28 1 solid
# remove stop words using anti_join() and the tidytext dataset of stop words
data(stop_words)

tidy_books <- tidy_books %>%
    anti_join(stop_words)

tidy_books
book linenumber chapter word
Sense & Sensibility 1 0 sense
Sense & Sensibility 1 0 sensibility
Sense & Sensibility 3 0 jane
Sense & Sensibility 3 0 austen
Sense & Sensibility 5 0 1811
Sense & Sensibility 10 1 chapter
Sense & Sensibility 10 1 1
Sense & Sensibility 13 1 family
Sense & Sensibility 13 1 dashwood
Sense & Sensibility 13 1 settled
Sense & Sensibility 13 1 sussex
Sense & Sensibility 13 1 estate
Sense & Sensibility 14 1 residence
Sense & Sensibility 14 1 norland
Sense & Sensibility 14 1 park
Sense & Sensibility 14 1 centre
Sense & Sensibility 15 1 property
Sense & Sensibility 15 1 generations
Sense & Sensibility 15 1 lived
Sense & Sensibility 16 1 respectable
Sense & Sensibility 16 1 manner
Sense & Sensibility 16 1 engage
Sense & Sensibility 16 1 opinion
Sense & Sensibility 17 1 surrounding
Sense & Sensibility 17 1 acquaintance
Sense & Sensibility 17 1 late
Sense & Sensibility 17 1 owner
Sense & Sensibility 17 1 estate
Sense & Sensibility 17 1 single
Sense & Sensibility 18 1 lived
Sense & Sensibility 18 1 advanced
Sense & Sensibility 18 1 age
Sense & Sensibility 19 1 life
Sense & Sensibility 19 1 constant
Sense & Sensibility 19 1 companion
Sense & Sensibility 19 1 housekeeper
Sense & Sensibility 19 1 sister
Sense & Sensibility 20 1 death
Sense & Sensibility 20 1 happened
Sense & Sensibility 20 1 ten
Sense & Sensibility 20 1 produced
Sense & Sensibility 21 1 alteration
Sense & Sensibility 21 1 home
Sense & Sensibility 21 1 supply
Sense & Sensibility 21 1 loss
Sense & Sensibility 21 1 invited
Sense & Sensibility 21 1 received
Sense & Sensibility 22 1 house
Sense & Sensibility 22 1 family
Sense & Sensibility 22 1 nephew
Sense & Sensibility 22 1 henry
Sense & Sensibility 22 1 dashwood
Sense & Sensibility 22 1 legal
Sense & Sensibility 23 1 inheritor
Sense & Sensibility 23 1 norland
Sense & Sensibility 23 1 estate
Sense & Sensibility 23 1 person
Sense & Sensibility 23 1 intended
Sense & Sensibility 24 1 bequeath
Sense & Sensibility 24 1 society
Sense & Sensibility 24 1 nephew
Sense & Sensibility 24 1 niece
Sense & Sensibility 25 1 children
Sense & Sensibility 25 1 gentleman’s
Sense & Sensibility 25 1 days
Sense & Sensibility 25 1 comfortably
Sense & Sensibility 25 1 spent
Sense & Sensibility 26 1 attachment
Sense & Sensibility 26 1 increased
Sense & Sensibility 26 1 constant
Sense & Sensibility 26 1 attention
Sense & Sensibility 27 1 henry
Sense & Sensibility 27 1 dashwood
Sense & Sensibility 27 1 wishes
Sense & Sensibility 27 1 proceeded
Sense & Sensibility 28 1 goodness
Sense & Sensibility 28 1 heart
Sense & Sensibility 28 1 degree
Sense & Sensibility 28 1 solid
Sense & Sensibility 29 1 comfort
Sense & Sensibility 29 1 age
Sense & Sensibility 29 1 receive
Sense & Sensibility 29 1 cheerfulness
Sense & Sensibility 30 1 children
Sense & Sensibility 30 1 added
Sense & Sensibility 30 1 relish
Sense & Sensibility 30 1 existence
Sense & Sensibility 32 1 marriage
Sense & Sensibility 32 1 henry
Sense & Sensibility 32 1 dashwood
Sense & Sensibility 32 1 son
Sense & Sensibility 33 1 lady
Sense & Sensibility 33 1 daughters
Sense & Sensibility 33 1 son
Sense & Sensibility 33 1 steady
Sense & Sensibility 33 1 respectable
Sense & Sensibility 34 1 amply
Sense & Sensibility 34 1 provided
Sense & Sensibility 34 1 fortune
Sense & Sensibility 34 1 mother
Sense & Sensibility 35 1 half
Sense & Sensibility 35 1 devolved
Sense & Sensibility 35 1 coming
Sense & Sensibility 35 1 age
Sense & Sensibility 36 1 marriage
Sense & Sensibility 36 1 likewise
Sense & Sensibility 36 1 happened
Sense & Sensibility 36 1 added
Sense & Sensibility 37 1 wealth
Sense & Sensibility 37 1 succession
Sense & Sensibility 37 1 norland
Sense & Sensibility 37 1 estate
Sense & Sensibility 38 1 sisters
Sense & Sensibility 38 1 fortune
Sense & Sensibility 38 1 independent
Sense & Sensibility 39 1 arise
Sense & Sensibility 39 1 father’s
Sense & Sensibility 39 1 inheriting
Sense & Sensibility 40 1 property
Sense & Sensibility 40 1 mother
Sense & Sensibility 41 1 father
Sense & Sensibility 41 1 thousand
Sense & Sensibility 41 1 pounds
Sense & Sensibility 41 1 disposal
Sense & Sensibility 42 1 remaining
Sense & Sensibility 42 1 moiety
Sense & Sensibility 42 1 wife’s
Sense & Sensibility 42 1 fortune
Sense & Sensibility 42 1 secured
Sense & Sensibility 43 1 child
Sense & Sensibility 43 1 life
Sense & Sensibility 45 1 gentleman
Sense & Sensibility 45 1 died
Sense & Sensibility 45 1 read
Sense & Sensibility 46 1 disappointment
Sense & Sensibility 46 1 pleasure
Sense & Sensibility 47 1 unjust
Sense & Sensibility 47 1 ungrateful
Sense & Sensibility 47 1 leave
Sense & Sensibility 47 1 estate
Sense & Sensibility 47 1 nephew
Sense & Sensibility 48 1 left
Sense & Sensibility 48 1 terms
Sense & Sensibility 48 1 destroyed
Sense & Sensibility 48 1 half
Sense & Sensibility 49 1 bequest
Sense & Sensibility 49 1 dashwood
Sense & Sensibility 49 1 wished
Sense & Sensibility 49 1 sake
Sense & Sensibility 49 1 wife
Sense & Sensibility 50 1 daughters
Sense & Sensibility 50 1 son
Sense & Sensibility 50 1 son
Sense & Sensibility 51 1 son’s
Sense & Sensibility 51 1 son
Sense & Sensibility 51 1 child
Sense & Sensibility 51 1 secured
Sense & Sensibility 52 1 leave
Sense & Sensibility 52 1 power
Sense & Sensibility 52 1 providing
Sense & Sensibility 52 1 dear
Sense & Sensibility 53 1 provision
Sense & Sensibility 53 1 charge
Sense & Sensibility 53 1 estate
Sense & Sensibility 54 1 sale
Sense & Sensibility 54 1 valuable
Sense & Sensibility 54 1 woods
Sense & Sensibility 54 1 tied
Sense & Sensibility 55 1 benefit
Sense & Sensibility 55 1 child
Sense & Sensibility 55 1 occasional
Sense & Sensibility 55 1 visits
Sense & Sensibility 55 1 father
Sense & Sensibility 56 1 mother
Sense & Sensibility 56 1 norland
Sense & Sensibility 56 1 gained
Sense & Sensibility 56 1 affections
Sense & Sensibility 56 1 uncle
Sense & Sensibility 57 1 attractions
Sense & Sensibility 57 1 means
Sense & Sensibility 57 1 unusual
Sense & Sensibility 57 1 children
Sense & Sensibility 58 1 imperfect
Sense & Sensibility 58 1 articulation
Sense & Sensibility 58 1 earnest
Sense & Sensibility 58 1 desire
Sense & Sensibility 59 1 cunning
Sense & Sensibility 59 1 tricks
Sense & Sensibility 59 1 deal
Sense & Sensibility 59 1 noise
Sense & Sensibility 59 1 outweigh
Sense & Sensibility 60 1 attention
Sense & Sensibility 60 1 received
Sense & Sensibility 61 1 niece
Sense & Sensibility 61 1 daughters
Sense & Sensibility 61 1 meant
Sense & Sensibility 61 1 unkind
Sense & Sensibility 62 1 mark
Sense & Sensibility 62 1 affection
Sense & Sensibility 62 1 girls
# use count() function to find the most common words in all the books as a
# whole
tidy_books %>%
    count(word, sort = TRUE)
word n
miss 1855
time 1337
fanny 862
dear 822
lady 817
sir 806
day 797
emma 787
sister 727
house 699
elizabeth 687
elinor 623
hope 601
friend 593
family 578
mind 563
father 558
home 550
jane 550
mother 541
catherine 539
feelings 535
happy 534
moment 515
half 501
love 495
till 495
crawford 493
marianne 492
heart 470
found 469
heard 468
anne 467
pleasure 462
morning 455
letter 450
poor 424
harriet 419
subject 418
brother 416
woman 416
world 410
left 407
cried 405
looked 404
feel 400
speak 395
evening 392
hear 392
weston 389
replied 387
manner 382
darcy 373
happiness 369
edmund 364
party 359
knightley 356
people 352
life 347
captain 342
told 339
opinion 337
spirits 334
suppose 329
acquaintance 328
immediately 323
friends 320
elton 319
ill 317
short 316
passed 314
leave 309
hour 308
idea 304
deal 303
eyes 302
word 298
attention 297
bennet 294
thomas 294
visit 294
colonel 292
comfort 292
coming 292
sort 292
return 287
brought 283
john 282
doubt 281
obliged 281
rest 281
answer 277
woodhouse 277
character 275
affection 272
chapter 272
minutes 272
perfectly 271
walk 264
aunt 263
glad 263
account 259
person 259
bingley 257
elliot 254
means 249
set 249
feeling 247
business 246
situation 245
wife 245
wished 245
hand 244
henry 244
days 243
talked 243
agreeable 242
pretty 242
added 241
town 241
bertram 240
daughter 240
received 240
talk 239
door 237
ladies 235
walked 235
carriage 233
conversation 233
object 233
body 232
dashwood 231
understand 231
manners 230
mary 229
spoke 229
reason 228
continued 227
called 226
ready 226
edward 225
head 223
london 223
returned 221
voice 220
bath 219
dare 219
determined 216
fine 216
sisters 216
impossible 215
marriage 214
regard 214
settled 213
married 211
read 211
fairfax 210
girl 210
care 209
frank 209
power 209
words 209
children 208
nature 208
change 204
expected 201
sense 201
country 200
kindness 200
jennings 199
talking 199
afraid 198
fortune 197
matter 197
appeared 196
giving 196
satisfied 196
thinking 196
tilney 196
directly 195
marry 195
stay 194
william 194
churchill 193
engaged 193
equal 193
assure 192
met 192
night 192
son 191
wentworth 191
believed 188
true 188
uncle 188
isabella 187
dinner 186
pleased 186
norris 185
rushworth 185
usual 185
# create visualization of most common words using ggplot2
library(ggplot2)

tidy_books %>%
    count(word, sort = TRUE) %>%
    filter(n > 600) %>%
    mutate(word = reorder(word, n)) %>%
    ggplot(aes(n, word)) + geom_col() + labs(y = NULL)

## The gutenbergr package

library(gutenbergr)

# Get text for The Time Machine, The War of the Worlds, The Invisible Man, and
# The Island of Doctor Moreau
hgwells <- gutenberg_download(c(35, 36, 5230, 159))

# Tokenize by word and remove stop words
tidy_hgwells <- hgwells %>%
    unnest_tokens(word, text) %>%
    anti_join(stop_words)

# Determine most common words using count() function
tidy_hgwells %>%
    count(word, sort = TRUE)
word n
time 461
people 302
door 260
heard 249
black 232
stood 229
white 224
hand 218
kemp 213
eyes 210
suddenly 210
found 200
invisible 200
night 200
day 189
montgomery 179
round 178
house 173
it’s 171
light 171
moment 171
head 166
hall 165
martians 165
looked 161
voice 155
strange 149
feet 147
left 147
mind 147
red 140
road 140
water 136
machine 133
presently 132
window 129
half 125
moreau 125
lay 123
human 118
hands 117
sat 116
air 115
trees 115
beast 114
marvel 114
don’t 113
fire 110
sky 108
hill 107
dark 106
running 105
green 104
passed 102
smoke 100
dead 99
i’m 99
coming 96
ran 96
brother 95
slowly 95
grew 93
stopped 92
world 92
days 91
fell 90
pit 90
street 90
told 87
sound 85
darkness 84
morning 84
body 83
ground 83
stared 83
staring 83
grey 81
london 81
houses 80
sea 80
martian 79
life 78
corner 77
common 76
fear 76
beach 75
creatures 75
glass 74
heat 74
idea 74
space 74
struck 74
appeared 73
blue 73
creature 73
story 73
sun 73
table 73
law 72
standing 72
blood 71
arm 70
earth 70
rose 70
cried 68
happened 68
pain 68
past 68
save 68
cylinder 67
food 67
island 67
close 66
minute 66
stranger 66
faint 65
empty 64
silent 64
remember 63
vanished 63
death 62
forward 62
huge 61
animal 60
crowd 60
he’s 60
scarcely 60
set 60
shoulder 60
that’s 60
understand 60
broken 59
caught 59
doubt 59
dog 58
figure 58
books 57
i’ve 57
sight 57
traveller 57
altogether 56
front 56
revolver 54
guns 53
heavy 53
hundred 53
mouth 53
woking 53
adye 52
curious 52
paper 52
watching 52
bright 51
can’t 51
foot 51
line 51
bar 50
escape 50
suppose 50
edge 49
eye 49
feeling 49
hot 49
remained 49
rushed 49
sudden 49
abruptly 48
bushes 48
dust 48
flung 48
imagine 48
matter 48
morlocks 48
nearer 48
terror 48
there’s 48
walked 48
bit 47
kitchen 47
rest 47
silence 47
station 47
direction 46
odd 46
smashed 46
weena 46
cut 45
hair 45
hard 45
sir 45
yards 45
afternoon 44
growing 44
headlong 44
lit 44
noise 44
quiet 44
shouting 44
bunting 43
country 43
# Get text for Jane Eyre, Wuthering Heights, The Tenant of Wildfell Hall,
# Villette, and Agnes Grey
bronte <- gutenberg_download(c(1260, 768, 969, 9182, 767))

tidy_bronte <- bronte %>%
    unnest_tokens(word, text) %>%
    anti_join(stop_words)

# Determine most common wrods using count() function
tidy_bronte %>%
    count(word, sort = TRUE)
word n
time 1065
miss 854
day 825
don’t 780
hand 767
eyes 714
night 648
heart 638
looked 601
door 591
till 584
half 583
house 580
love 565
life 560
heard 510
left 492
mind 492
sir 490
head 487
home 461
heathcliff 421
mother 417
moment 409
found 405
morning 405
leave 404
told 404
lady 403
i’ll 399
round 386
stood 374
i’m 366
replied 365
john 364
evening 362
jane 362
words 357
speak 356
answered 355
cried 349
hear 348
light 348
hour 346
linton 346
rose 344
voice 344
madame 343
child 339
catherine 338
hope 331
master 331
eye 330
sat 329
fire 328
word 318
rochester 317
father 316
god 309
feel 306
hands 305
mine 303
answer 300
graham 299
cold 294
world 293
poor 290
continued 288
dark 285
suppose 284
brought 280
passed 279
bed 275
pleasure 271
fear 270
care 268
woman 268
read 262
talk 262
return 259
strange 259
called 256
window 256
girl 255
quiet 255
friends 254
white 253
set 252
helen 251
days 245
rest 242
table 242
lay 241
papa 240
nature 239
smile 238
friend 237
returned 236
coming 235
matter 235
dr 234
happy 233
chair 232
dear 231
walk 231
entered 229
book 227
black 226
bretton 225
feelings 223
minutes 218
close 217
live 216
chapter 215
hall 213
remember 213
it’s 212
air 211
de 210
doubt 210
glad 210
huntingdon 209
deep 206
manner 206
people 206
presence 206
wished 206
hair 204
school 204
sense 202
strong 202
lips 201
sort 201
spoke 201
garden 200
fine 199
sister 198
call 197
feeling 197
true 197
ladies 196
stay 195
arthur 194
can’t 194
wife 194
death 193
sit 193
short 191
won’t 191
appeared 190
letter 190
truth 190
children 189
held 189
lucy 189
bear 188
idea 188
spirit 187
i’ve 186
son 184
person 183
st 183
silence 182
trouble 181
exclaimed 179
grey 177
low 177
business 176
sweet 175
company 174
added 172
mary 172
hard 171
power 171
question 171
tone 171
arm 170
times 170
hareton 169
mamma 167
paul 167
brother 166
ill 165
reason 165
character 162
fell 162
pale 162
pretty 162
arms 160
bright 160
he’s 160
past 160
suddenly 160
drawing 159
gentleman 159
step 159
struck 159
subject 159
water 159
bad 158
# Bind data frames for Jane Austen, Bronte sisters, and H.G. Wells
library(tidyr)

frequency <- bind_rows(mutate(tidy_bronte, author = "Brontë Sisters"), mutate(tidy_hgwells,
    author = "H.G. Wells"), mutate(tidy_books, author = "Jane Austen")) %>%
    mutate(word = str_extract(word, "[a-z']+")) %>%
    count(author, word) %>%
    group_by(author) %>%
    mutate(proportion = n/sum(n)) %>%
    select(-n) %>%
    pivot_wider(names_from = author, values_from = proportion) %>%
    pivot_longer(`Brontë Sisters`:`H.G. Wells`, names_to = "author", values_to = "proportion")

frequency
word Jane Austen author proportion
a 0 Brontë Sisters 0
a 0 H.G. Wells 0
aback NA Brontë Sisters 0
aback NA H.G. Wells 0
abaht NA Brontë Sisters 0
abaht NA H.G. Wells NA
abandon NA Brontë Sisters 0
abandon NA H.G. Wells 0
abandoned 0 Brontë Sisters 0
abandoned 0 H.G. Wells 0
abandoning NA Brontë Sisters 0
abandoning NA H.G. Wells 0
abandonment NA Brontë Sisters 0
abandonment NA H.G. Wells 0
abase NA Brontë Sisters 0
abase NA H.G. Wells NA
abasement NA Brontë Sisters 0
abasement NA H.G. Wells NA
abashed 0 Brontë Sisters 0
abashed 0 H.G. Wells NA
abate 0 Brontë Sisters 0
abate 0 H.G. Wells NA
abated NA Brontë Sisters 0
abated NA H.G. Wells NA
abating 0 Brontë Sisters 0
abating 0 H.G. Wells NA
abbot NA Brontë Sisters 0
abbot NA H.G. Wells NA
abbreviation 0 Brontë Sisters 0
abbreviation 0 H.G. Wells NA
abc NA Brontë Sisters 0
abc NA H.G. Wells NA
abdicated NA Brontë Sisters 0
abdicated NA H.G. Wells NA
abduction NA Brontë Sisters 0
abduction NA H.G. Wells NA
abed NA Brontë Sisters 0
abed NA H.G. Wells NA
aberdeen NA Brontë Sisters 0
aberdeen NA H.G. Wells NA
aberration NA Brontë Sisters 0
aberration NA H.G. Wells NA
abetted NA Brontë Sisters 0
abetted NA H.G. Wells NA
abhor 0 Brontë Sisters 0
abhor 0 H.G. Wells NA
abhorred 0 Brontë Sisters 0
abhorred 0 H.G. Wells NA
abhorrence 0 Brontë Sisters 0
abhorrence 0 H.G. Wells 0
abhors NA Brontë Sisters 0
abhors NA H.G. Wells NA
abide 0 Brontë Sisters 0
abide 0 H.G. Wells 0
abigail NA Brontë Sisters 0
abigail NA H.G. Wells NA
abigails NA Brontë Sisters 0
abigails NA H.G. Wells NA
abilities 0 Brontë Sisters 0
abilities 0 H.G. Wells NA
ability 0 Brontë Sisters 0
ability 0 H.G. Wells 0
abject NA Brontë Sisters 0
abject NA H.G. Wells 0
abjured NA Brontë Sisters 0
abjured NA H.G. Wells NA
ablaze NA Brontë Sisters 0
ablaze NA H.G. Wells 0
ablutions NA Brontë Sisters 0
ablutions NA H.G. Wells NA
ably NA Brontë Sisters 0
ably NA H.G. Wells NA
abnormal NA Brontë Sisters 0
abnormal NA H.G. Wells 0
abode 0 Brontë Sisters 0
abode 0 H.G. Wells NA
abodes NA Brontë Sisters 0
abodes NA H.G. Wells NA
abominable 0 Brontë Sisters 0
abominable 0 H.G. Wells 0
abominably 0 Brontë Sisters 0
abominably 0 H.G. Wells NA
aboon NA Brontë Sisters 0
aboon NA H.G. Wells NA
aboriginal NA Brontë Sisters 0
aboriginal NA H.G. Wells NA
abortive NA Brontë Sisters 0
abortive NA H.G. Wells NA
abounded 0 Brontë Sisters 0
abounded 0 H.G. Wells 0
abounding 0 Brontë Sisters 0
abounding 0 H.G. Wells NA
abridge 0 Brontë Sisters 0
abridge 0 H.G. Wells NA
abridged NA Brontë Sisters 0
abridged NA H.G. Wells NA
abridgment NA Brontë Sisters 0
abridgment NA H.G. Wells NA
abroad 0 Brontë Sisters 0
abroad 0 H.G. Wells 0
abrupt 0 Brontë Sisters 0
abrupt 0 H.G. Wells 0
abruptly 0 Brontë Sisters 0
abruptly 0 H.G. Wells 0
abruptness 0 Brontë Sisters 0
abruptness 0 H.G. Wells 0
absalom NA Brontë Sisters 0
absalom NA H.G. Wells NA
absconded NA Brontë Sisters 0
absconded NA H.G. Wells NA
absence 0 Brontë Sisters 0
absence 0 H.G. Wells 0
absences 0 Brontë Sisters 0
absences 0 H.G. Wells NA
absent 0 Brontë Sisters 0
absent 0 H.G. Wells 0
absented 0 Brontë Sisters 0
absented 0 H.G. Wells NA
absently NA Brontë Sisters 0
absently NA H.G. Wells NA
absolute 0 Brontë Sisters 0
absolute 0 H.G. Wells 0
absolutely 0 Brontë Sisters 0
absolutely 0 H.G. Wells 0
absolutism NA Brontë Sisters 0
absolutism NA H.G. Wells NA
absolve NA Brontë Sisters 0
absolve NA H.G. Wells NA
absolved NA Brontë Sisters 0
absolved NA H.G. Wells NA
absolves NA Brontë Sisters 0
absolves NA H.G. Wells NA
absorb NA Brontë Sisters 0
absorb NA H.G. Wells 0
absorbed 0 Brontë Sisters 0
absorbed 0 H.G. Wells 0
absorbing 0 Brontë Sisters 0
absorbing 0 H.G. Wells NA
abstain 0 Brontë Sisters 0
abstain 0 H.G. Wells 0
abstained 0 Brontë Sisters 0
abstained 0 H.G. Wells NA
abstaining NA Brontë Sisters 0
abstaining NA H.G. Wells NA
abstemious NA Brontë Sisters 0
abstemious NA H.G. Wells NA
abstemiousness NA Brontë Sisters 0
abstemiousness NA H.G. Wells NA
abstinence 0 Brontë Sisters 0
abstinence 0 H.G. Wells NA
abstract 0 Brontë Sisters 0
abstract 0 H.G. Wells 0
abstracted 0 Brontë Sisters 0
abstracted 0 H.G. Wells NA
abstractedly NA Brontë Sisters 0
abstractedly NA H.G. Wells NA
abstracting NA Brontë Sisters 0
abstracting NA H.G. Wells NA
abstraction 0 Brontë Sisters 0
abstraction 0 H.G. Wells 0
abstractions NA Brontë Sisters 0
abstractions NA H.G. Wells 0
absurd 0 Brontë Sisters 0
absurd 0 H.G. Wells 0
absurdities 0 Brontë Sisters 0
absurdities 0 H.G. Wells NA
absurdity 0 Brontë Sisters 0
absurdity 0 H.G. Wells 0
absurdly NA Brontë Sisters 0
absurdly NA H.G. Wells 0
abundance 0 Brontë Sisters 0
abundance 0 H.G. Wells 0
abundant 0 Brontë Sisters 0
abundant 0 H.G. Wells 0
abundantly 0 Brontë Sisters 0
abundantly 0 H.G. Wells NA
abuse 0 Brontë Sisters 0
abuse 0 H.G. Wells 0
abused 0 Brontë Sisters 0
abused 0 H.G. Wells 0
abuses 0 Brontë Sisters 0
abuses 0 H.G. Wells NA
abusing 0 Brontë Sisters 0
abusing 0 H.G. Wells NA
abusive 0 Brontë Sisters 0
abusive 0 H.G. Wells NA
abyss 0 Brontë Sisters 0
abyss 0 H.G. Wells NA
acacia 0 Brontë Sisters 0
acacia 0 H.G. Wells NA
academician NA Brontë Sisters 0
academician NA H.G. Wells NA
academicians NA Brontë Sisters 0
academicians NA H.G. Wells NA
accede 0 Brontë Sisters 0
accede 0 H.G. Wells NA
acceded 0 Brontë Sisters 0
acceded 0 H.G. Wells NA
accelerated NA Brontë Sisters 0
accelerated NA H.G. Wells 0
# plot findings
library(scales)

ggplot(frequency, aes(x = proportion, y = `Jane Austen`, color = abs(`Jane Austen` -
    proportion))) + geom_abline(color = "gray40", lty = 2) + geom_jitter(alpha = 0.1,
    size = 2.5, width = 0.3, height = 0.3) + geom_text(aes(label = word), check_overlap = TRUE,
    vjust = 1.5) + scale_x_log10(labels = percent_format()) + scale_y_log10(labels = percent_format()) +
    scale_color_gradient(limits = c(0, 0.001), low = "darkslategray4", high = "gray75") +
    facet_wrap(~author, ncol = 2) + theme(legend.position = "none") + labs(y = "Jane Austen",
    x = NULL)

# correlation test to determine how similar and different sets of word
# frequencies are
cor.test(data = frequency[frequency$author == "Brontë Sisters", ], ~proportion +
    `Jane Austen`)
## 
##  Pearson's product-moment correlation
## 
## data:  proportion and Jane Austen
## t = 110.73, df = 10275, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  0.7286645 0.7462983
## sample estimates:
##       cor 
## 0.7376071
cor.test(data = frequency[frequency$author == "H.G. Wells", ], ~proportion + `Jane Austen`)
## 
##  Pearson's product-moment correlation
## 
## data:  proportion and Jane Austen
## t = 35.229, df = 6008, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  0.3925925 0.4345058
## sample estimates:
##       cor 
## 0.4137684

In-Class Practice

My First Function - I will create a function that adds two numbers.

# create function to add two numbers
myfunc <- function(x, y) {
    z <- x + y
    return(z)
}

# run function to add 1 + 9
myfunc(1, 9)
## [1] 10

Update My First Function - Now I will update the function to take the mean of two numbers.

# change function to compute the mean between two numbers
myfunc <- function(x, y, v) {
    z <- (x + y)/v
    return(z)
}

# run function to take average of 1 and 9
myfunc(1, 9, 2)
## [1] 5
# try again
myfunc_revised <- function(...) {
    args <- list(...)
    z <- sum(...)/length(args)
    return(z)
}

myfunc_revised(1, 9)
## [1] 5
myfunc_revised(1, 1, 7)
## [1] 3

In-Class Challenge!

Word Removal

I am going to create a function to remove a word from a string. Then, I will test the function by trying to remove the word “Mary” from “Mary had a little lamb”

# create function to remove 'word' from 'mystring' and return the updated
# string
remove_word_func <- function(mystring, word) {
    x <- mystring
    z <- word
    y <- gsub(z, "", x)
    return(y)
}

# run function to remove 'Mary' from 'Mary had a little lamb'
remove_word_func("Mary had a little lamb", "Mary")
## [1] " had a little lamb"

Punctuation Removal

Now, I am going to create a function that removes punctuation from a string. Then, I will test the function by trying to remove “?!?!” from “Mary had a little lamb?!?!”

# create function to remove punctuation from 'mystring' and return the updated
# string
remove_punc_func <- function(mystring) {
    x <- mystring
    y <- gsub("[[:punct:]]", "", x)
    return(y)
}

# run function to remove '?!?!' from 'Mary had a little lamb?!?!'
remove_punc_func("Mary had a little lamb?!?!")
## [1] "Mary had a little lamb"

Level Up

First I am going to remove non-word tokens (aka punctuation) from a string. Then I am going to split the string to one word per row.

Remove non-word tokens using previous remove_punc_func function

# use remove_punc_func to remove punctuation from 'Mary had a little lamb?!?!')
# and call this new string marydat
marydat <- remove_punc_func("Mary had a little lamb?!?!")
marydat
## [1] "Mary had a little lamb"

Split string to one word per row

# create data frame that takes marydat string and splits into one word per row
df <- data.frame(word = unlist(strsplit(marydat, "\\s+")))

# print data frame
print(df)
##     word
## 1   Mary
## 2    had
## 3      a
## 4 little
## 5   lamb
# try using textbook method
mary_df <- tibble(text = marydat)
mary_df %>%
    unnest_tokens(word, text)
word
mary
had
a
little
lamb

Additional Class Stuff

Importing Text into R

Plain text (txt) and csv files are your go-to formats. There are times when you will want to retain metadata (document titles, dates, etc.) for multidocument corpora.

Local Txt File

From your machine’s ‘data’ folder inside of this project folder. The problem with this is of course that another user can’t knit your markdown unless they install

mytext1 <- readLines("data/pangram.txt")
print(mytext1)
## [1] "The quick brown fox jumped over the lazy dog."

Local CSV File

CSV file with some meta-data appended

mytext2 <- read.csv("data/pantext.csv")
print(mytext2)
##      doc_id
## 1 MySample1
##                                                                               doc_text
## 1 The quick brown fox jumps over the lazy dog. Jack and Jill went up hill. I see cats.

URL Public File How To

Store your text file in a public repository (e.g., your github repo). Make sure that you have the rights to share the text. Copy the URL for the raw file.

Screenshot of Github Repository

# read txt file from github, paste all lines from text into one continuous file
# using ' ' as separator
sherlock <- paste(readLines("https://raw.githubusercontent.com/Reilly-ConceptsCognitionLab/reillylab_publicdata/main/SherlockHolmes.txt"))

# Print the first few lines to check content
cat(sherlock[1:5], sep = "\n")
## To Sherlock Holmes she is always _the_ woman. I have seldom heard him
## mention her under any other name. In his eyes she eclipses and
## predominates the whole of her sex. It was not that he felt any emotion
## akin to love for Irene Adler. All emotions, and that one particularly,
## were abhorrent to his cold, precise but admirably balanced mind. He

Import Unabomber

Import and inspect the raw unabomber manifesto, inspect

unabomb <- paste(readLines("https://raw.githubusercontent.com/Reilly-ConceptsCognitionLab/reillylab_publicdata/main/unabomber_manifesto.txt"))

# Print first few lines of unabomber manifesto
cat(unabomb[1:10], sep = "\n")
## INDUSTRIAL SOCIETY AND ITS FUTURE
## 
## Introduction
## 
## 1. The Industrial Revolution and its consequences have been a disaster for the 
##    human race. They have greatly increased the life-expectancy of those of us 
##    who live in “advanced” countries, but they have destabilized society, have 
##    made life unfulfilling, have subjected human beings to indignities, have led 
##    to widespread psychological suffering (in the Third World to physical 
##    suffering as well) and have inflicted severe damage on the natural world. The



Exporting from R

Sometimes you want to save a text sample to your own machine. You can save it in raw plain text format or RDA format (a compressed format that R likes and will keep you under file size limits)

# create a new dataframe
mysample <- c("Mary had a little lamb")
print(mysample)
## [1] "Mary had a little lamb"
# write that new dataframe to your hard drive as a CSV file
write.csv(mysample, "scratch/mysample.csv", row.names = F)

# Saving text in a compressed format R likes
save(mysample, file = "scratch/mysample.rda")



Importing BIG data

We’ll look at the Jane Austen package and then mess with Project Gutenberg

# if regular installation doesn't work try this: install.packages('gutenbergr',
# type = 'source')

# Load libraries
library(gutenbergr)
library(janeaustenr)



gutenbergr

visit project gutenberg
Project Gutenberg contains >60,000 free works in many different formats, languages, and file types. What we want for text mining is plain text. More specifically, we want structured text files. Files don’t come automatically prepped that way from Project Gutenberg, but we can structure them ourselves. Per the Gutenberg website:

“Project Gutenberg was the first provider of free electronic books, or eBooks. Michael Hart, founder of Project Gutenberg, invented eBooks in 1971 and his memory continues to inspire the creation of eBooks and related content today.”– Return specific criteria:

gutenberg_works(..., languages = "en", only_text = TRUE, rights = c("Public domain in the USA.", "None"), distinct = TRUE, all_languages = FALSE, only_languages = TRUE)

Filter

glimpse(gutenberg_works()) lists all English language texts in project Gutenberg by whatever metafield you would like to filter on

gutenberg_works(author == "Dickens, Charles", languages = "en", only_text = TRUE,
    rights = NULL, distinct = TRUE, all_languages = FALSE, only_languages = TRUE)  #all english works regardless of copyright
gutenberg_id title author gutenberg_author_id language gutenberg_bookshelf rights has_text
46 A Christmas Carol in Prose; Being a Ghost Story of Christmas Dickens, Charles 37 en Children’s Literature/Christmas/Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
98 A Tale of Two Cities Dickens, Charles 37 en Historical Fiction/Category: Historical Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
564 The Mystery of Edwin Drood Dickens, Charles 37 en Mystery Fiction/Category: Crime, Thrillers and Mystery/Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
580 The Pickwick Papers Dickens, Charles 37 en Best Books Ever Listings/Category: Humour/Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
588 Master Humphrey’s Clock Dickens, Charles 37 en Category: Novels/Category: British Literature Public domain in the USA. TRUE
644 The Haunted Man and the Ghost’s Bargain Dickens, Charles 37 en Christmas/Category: Novels/Category: British Literature Public domain in the USA. TRUE
650 Pictures from Italy Dickens, Charles 37 en Category: Travel Writing/Category: British Literature Public domain in the USA. TRUE
653 The Chimes A Goblin Story of Some Bells That Rang an Old Year out and a New Year In Dickens, Charles 37 en Category: Novels/Category: British Literature Public domain in the USA. TRUE
675 American Notes Dickens, Charles 37 en Category: Travel Writing/Category: British Literature Public domain in the USA. TRUE
676 The Battle of Life Dickens, Charles 37 en Christmas/Category: Novels/Category: British Literature Public domain in the USA. TRUE
678 The Cricket on the Hearth: A Fairy Tale of Home Dickens, Charles 37 en Children’s Literature/Category: Novels/Category: British Literature Public domain in the USA. TRUE
699 A Child’s History of England Dickens, Charles 37 en Children’s History/United Kingdom/Category: Children & Young Adult Reading/Category: History - British Public domain in the USA. TRUE
700 The Old Curiosity Shop Dickens, Charles 37 en Category: Novels/Category: British Literature Public domain in the USA. TRUE
730 Oliver Twist Dickens, Charles 37 en Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
766 David Copperfield Dickens, Charles 37 en Harvard Classics/Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
786 Hard Times Dickens, Charles 37 en Category: Novels/Category: British Literature Public domain in the USA. TRUE
807 Hunted Down: The Detective Stories of Charles Dickens Dickens, Charles 37 en Detective Fiction/Category: Short Stories/Category: Crime, Thrillers and Mystery/Category: British Literature Public domain in the USA. TRUE
809 Holiday Romance Dickens, Charles 37 en Children’s Picture Books/Category: Short Stories/Category: Children & Young Adult Reading/Category: Humour/Category: British Literature Public domain in the USA. TRUE
810 George Silverman’s Explanation Dickens, Charles 37 en Category: Novels/Category: British Literature Public domain in the USA. TRUE
821 Dombey and Son Dickens, Charles 37 en Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
824 Speeches: Literary and Social Dickens, Charles 37 en Category: Essays, Letters & Speeches/Category: British Literature Public domain in the USA. TRUE
872 Reprinted Pieces Dickens, Charles 37 en Category: Short Stories/Category: Essays, Letters & Speeches/Category: Travel Writing/Category: British Literature Public domain in the USA. TRUE
882 Sketches by Boz, Illustrative of Every-Day Life and Every-Day People Dickens, Charles 37 en Category: Short Stories/Category: British Literature Public domain in the USA. TRUE
883 Our Mutual Friend Dickens, Charles 37 en Category: Novels/Category: British Literature Public domain in the USA. TRUE
888 The Lazy Tour of Two Idle Apprentices Dickens, Charles 37 en Category: Humour/Category: Travel Writing/Category: British Literature Public domain in the USA. TRUE
912 Mudfog and Other Sketches Dickens, Charles 37 en Category: Short Stories/Category: Humour/Category: British Literature Public domain in the USA. TRUE
914 The Uncommercial Traveller Dickens, Charles 37 en Category: Essays, Letters & Speeches/Category: Travel Writing/Category: British Literature Public domain in the USA. TRUE
916 Sketches of Young Couples Dickens, Charles 37 en Category: Humour/Category: Essays, Letters & Speeches/Category: British Literature Public domain in the USA. TRUE
917 Barnaby Rudge: A Tale of the Riots of ’Eighty Dickens, Charles 37 en Historical Fiction/Category: Historical Novels/Category: Novels/Category: British Literature Public domain in the USA. TRUE
918 Sketches of Young Gentlemen Dickens, Charles 37 en Category: Humour/Category: British Literature Public domain in the USA. TRUE
922 Sunday Under Three Heads Dickens, Charles 37 en Category: Essays, Letters & Speeches/Category: British Literature Public domain in the USA. TRUE
924 To Be Read at Dusk Dickens, Charles 37 en Category: Short Stories/Category: Crime, Thrillers and Mystery/Category: British Literature Public domain in the USA. TRUE
927 The Lamplighter Dickens, Charles 37 en Category: Humour/Category: Novels/Category: British Literature Public domain in the USA. TRUE
963 Little Dorrit Dickens, Charles 37 en Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
967 Nicholas Nickleby Dickens, Charles 37 en Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
968 Martin Chuzzlewit Dickens, Charles 37 en Best Books Ever Listings/Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
1023 Bleak House Dickens, Charles 37 en Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
1289 Three Ghost Stories Dickens, Charles 37 en Category: Short Stories/Category: Humour/Category: British Literature Public domain in the USA. TRUE
1392 The Seven Poor Travellers Dickens, Charles 37 en Category: Short Stories/Category: British Literature Public domain in the USA. TRUE
1394 The Holly-Tree Dickens, Charles 37 en Category: Short Stories/Category: British Literature Public domain in the USA. TRUE
1400 Great Expectations Dickens, Charles 37 en Best Books Ever Listings/Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
1406 The Perils of Certain English Prisoners Dickens, Charles 37 en Category: Historical Novels/Category: Adventure/Category: Novels/Category: British Literature Public domain in the USA. TRUE
1407 A Message from the Sea Dickens, Charles 37 en Category: Short Stories/Category: British Literature Public domain in the USA. TRUE
1413 Tom Tiddler’s Ground Dickens, Charles 37 en Category: Short Stories/Category: British Literature Public domain in the USA. TRUE
1414 Somebody’s Luggage Dickens, Charles 37 en Category: Short Stories/Category: Humour/Category: British Literature Public domain in the USA. TRUE
1415 Doctor Marigold Dickens, Charles 37 en Category: Novels/Category: British Literature Public domain in the USA. TRUE
1416 Mrs. Lirriper’s Lodgings Dickens, Charles 37 en Category: Novels/Category: British Literature Public domain in the USA. TRUE
1419 Mugby Junction Dickens, Charles 37 en Category: Short Stories/Category: British Literature Public domain in the USA. TRUE
1421 Mrs. Lirriper’s Legacy Dickens, Charles 37 en Category: Novels/Category: British Literature Public domain in the USA. TRUE
1422 Going into Society Dickens, Charles 37 en Category: Short Stories/Category: Humour/Category: British Literature Public domain in the USA. TRUE
1423 No Thoroughfare Dickens, Charles 37 en Category: Crime, Thrillers and Mystery/Category: Novels/Category: British Literature Public domain in the USA. TRUE
1435 Miscellaneous Papers Dickens, Charles 37 en Category: Essays, Letters & Speeches/Category: British Literature Public domain in the USA. TRUE
1465 The Wreck of the Golden Mary Dickens, Charles 37 en Category: Adventure/Category: Novels/Category: British Literature Public domain in the USA. TRUE
1467 Some Christmas Stories Dickens, Charles 37 en Christmas/Category: Short Stories/Category: British Literature Public domain in the USA. TRUE
2324 A House to Let Dickens, Charles 37 en Category: Short Stories/Category: Crime, Thrillers and Mystery/Category: British Literature Public domain in the USA. TRUE
9691 All the Year Round Dickens, Charles 37 en Copyrighted. Read the copyright notice inside this book for details. TRUE
9696 A Christmas Carol Dickens, Charles 37 en Copyrighted. Read the copyright notice inside this book for details. TRUE
9712 Hunted Down Dickens, Charles 37 en Detective Fiction Copyrighted. Read the copyright notice inside this book for details. TRUE
9729 Perils of Certain English Prisoners Dickens, Charles 37 en Copyrighted. Read the copyright notice inside this book for details. TRUE
9732 Barnaby Rudge Dickens, Charles 37 en Historical Fiction Copyrighted. Read the copyright notice inside this book for details. TRUE
9733 Sketches by Boz, illustrative of everyday life and every-day people Dickens, Charles 37 en Copyrighted. Read the copyright notice inside this book for details. TRUE
9736 Sunday under Three Heads Dickens, Charles 37 en Copyrighted. Read the copyright notice inside this book for details. TRUE
9739 Cricket on the Hearth Dickens, Charles 37 en Children’s Literature Copyrighted. Read the copyright notice inside this book for details. TRUE
15618 The Loving Ballad of Lord Bateman Dickens, Charles 37 en Category: Poetry/Category: Humour/Category: Mythology, Legends & Folklore/Category: British Literature Public domain in the USA. TRUE
20795 The Cricket on the Hearth Dickens, Charles 37 en Children’s Literature/Category: Novels/Category: British Literature Public domain in the USA. TRUE
23344 The Magic Fishbone A Holiday Romance from the Pen of Miss Alice Rainbird, Aged 7 Dickens, Charles 37 en Children’s Picture Books/Category: Children & Young Adult Reading/Category: Science-Fiction & Fantasy/Category: British Literature Public domain in the USA. TRUE
23452 The Trial of William Tinkling Written by Himself at the Age of 8 Years Dickens, Charles 37 en Children’s Picture Books/Category: Children & Young Adult Reading/Category: Humour/Category: Adventure/Category: British Literature Public domain in the USA. TRUE
23765 Captain Boldheart & the Latin-Grammar Master A Holiday Romance from the Pen of Lieut-Col. Robin Redforth, aged 9 Dickens, Charles 37 en Children’s Picture Books/Category: Children & Young Adult Reading/Category: Humour/Category: Adventure/Category: British Literature Public domain in the USA. TRUE
25852 The Letters of Charles Dickens. Vol. 1, 1833-1856 Dickens, Charles 37 en Category: Biographies/Category: Essays, Letters & Speeches/Category: British Literature Public domain in the USA. TRUE
25853 The Letters of Charles Dickens. Vol. 2, 1857-1870 Dickens, Charles 37 en Category: Biographies/Category: Essays, Letters & Speeches/Category: British Literature Public domain in the USA. TRUE
25854 The Letters of Charles Dickens. Vol. 3, 1836-1870 Dickens, Charles 37 en Category: Biographies/Category: Essays, Letters & Speeches/Category: British Literature Public domain in the USA. TRUE
25985 Bardell v. Pickwick Dickens, Charles 37 en Category: Humour/Category: Novels/Category: British Literature Public domain in the USA. TRUE
30127 Tales from Dickens Dickens, Charles 37 en Category: Short Stories/Category: British Literature Public domain in the USA. TRUE
30368 A Christmas Carol The original manuscript Dickens, Charles 37 en Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
32241 Dickens’ Stories About Children Every Child Can Read Dickens, Charles 37 en Category: Short Stories/Category: Children & Young Adult Reading/Category: British Literature Public domain in the USA. TRUE
35536 The Poems and Verses of Charles Dickens Dickens, Charles 37 en Category: Poetry/Category: British Literature Public domain in the USA. TRUE
37121 Charles Dickens’ Children Stories Dickens, Charles 37 en Category: Short Stories/Category: Children & Young Adult Reading/Category: British Literature Public domain in the USA. TRUE
40723 The Battle of Life: A Love Story Dickens, Charles 37 en Category: Novels/Category: British Literature Public domain in the USA. TRUE
40729 “Old Scrooge”: A Christmas Carol in Five Staves. Dramatized from Charles Dickens’ Celebrated Christmas Story. Dickens, Charles 37 en Category: Plays/Films/Dramas/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
41739 A Christmas Carol; Or, The Miser’s Warning! (Adapted from Charles Dickens’ Celebrated Work.) Dickens, Charles 37 en Category: Plays/Films/Dramas/Category: British Literature Public domain in the USA. TRUE
41894 Christmas-Tide Dickens, Charles 37 en Category: Children & Young Adult Reading/Category: Essays, Letters & Speeches/Category: Parenthood & Family Relations/Category: Teaching & Education Public domain in the USA. TRUE
42232 A Child’s Dream of a Star Dickens, Charles 37 en Category: Short Stories/Category: British Literature Public domain in the USA. TRUE
43111 The Personal History of David Copperfield Dickens, Charles 37 en Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
43207 Scenes and Characters from the Works of Charles Dickens Being Eight Hundred and Sixty-six Pictures Printed from the Original Wood Blocks Dickens, Charles 37 en Category: British Literature/Category: Art Public domain in the USA. TRUE
46675 Oliver Twist; or, The Parish Boy’s Progress. Illustrated Dickens, Charles 37 en Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
47529 Oliver Twist, Vol. 1 (of 3) Dickens, Charles 37 en Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
47530 Oliver Twist, Vol. 2 (of 3) Dickens, Charles 37 en Category: Crime, Thrillers and Mystery/Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
47531 Oliver Twist, Vol. 3 (of 3) Dickens, Charles 37 en Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
47534 The Posthumous Papers of the Pickwick Club, v. 1 (of 2) Dickens, Charles 37 en Category: Humour/Category: Novels/Category: British Literature Public domain in the USA. TRUE
47535 The Posthumous Papers of the Pickwick Club, v. 2 (of 2) Dickens, Charles 37 en Category: Humour/Category: Novels/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
49125 Stories from Dickens Dickens, Charles 37 en Category: Short Stories/Category: Children & Young Adult Reading/Category: British Literature Public domain in the USA. TRUE
49683 Cruikshank’s Water Colours Dickens, Charles 37 en Category: British Literature/Category: Art Public domain in the USA. TRUE
49927 Pearl-Fishing; Choice Stories from Dickens’ Household Words; First Series Dickens, Charles 37 en Category: Short Stories/Category: British Literature Public domain in the USA. TRUE
50334 Pearl-Fishing; Choice Stories from Dickens’ Household Words; Second Series Dickens, Charles 37 en Category: Short Stories/Category: Classics of Literature/Category: British Literature Public domain in the USA. TRUE
52125 Nell and Her Grandfather, Told from Charles Dickens’s “The Old Curiosity Shop” Dickens, Charles 37 en Category: Novels/Category: British Literature Public domain in the USA. TRUE
58157 Index of the Project Gutenberg Works of Charles Dickens Dickens, Charles 37 en Category: Encyclopedias/Dictionaries/Reference Public domain in the USA. TRUE
61193 Mr. Pickwick’s Christmas Being an Account of the Pickwickians’ Christmas at the Manor Farm, of the Adventures There; the Tale of the Goblin Who Stole a Sexton, and of the Famous Sports on the Ice Dickens, Charles 37 en Category: Humour/Category: Novels/Category: British Literature Public domain in the USA. TRUE
61668 Stories of Intellect Dickens, Charles 37 en Category: Short Stories/Category: British Literature Public domain in the USA. TRUE
71858 Holly berries from Dickens Dickens, Charles 37 en Category: Essays, Letters & Speeches/Category: British Literature Public domain in the USA. TRUE
75856 The children of Dickens Dickens, Charles 37 en Category: Essays, Letters & Speeches/Category: British Literature Public domain in the USA. TRUE


Search Titles

library of congress subject headings (lcsh) “Library of Congress Subject Headings (LCSH) is perhaps the most widely adopted subject indexing language in the world, has been translated into many languages, and is used around the world by libraries large and small” library of congress classification (lcc)< Link and find codes by subject.

gutenberg_subjects %>%
    filter(subject_type == "lcsh") %>%
    count(subject, sort = TRUE)
sci <- gutenberg_subjects %>%
    filter(subject_type == "lcc", subject == "Q")  #return a vectorized list of all texts about 'science' just outputting their ID
extract <- as.vector(sci$gutenberg_id)  #writes gutenberg ids to a numeric vector
print(extract)
gutenberg_works(gutenberg_id == extract)  #return in index of all english works regardless of copyright about the subject type Q (which is the code for science)


Download 1 Doc

Download by gutenberg_id – here are some other fields:
gutenberg_download(gutenberg_id, mirror = NULL, strip = TRUE, meta_fields = TRUE, verbose = TRUE, files = NULL)

great <- gutenberg_download(1400, meta_fields = c("title", "author"))  #strip dumps stuff from the header, includes author, title, text in the dataframe
head(great)

Download multiple documents

add meta fields

# more.dickens <- gutenberg_download(c(588,644,1400), meta_fields = c('title',
# 'author')) head(more.dickens)

Download all works by one author

That are in English and have full text. Write the downloaded files to a folder then comment out so it doesn’t download them anew every time you knit.

# alldickens <- gutenberg_works(author == 'Dickens, Charles')) #isolate dickens
# works dickids <- alldickens$gutenberg_id #extract vector of document ids
# alldickens <- gutenberg_download(dickids, meta_fields = c('title', 'author'))
# str(alldickens)

janeaustenr

read about the jane austen package here
Fulltexts for Jane Austen’s 6 completed novels:
Sense & Sensibility
Pride & Prejudice
Mansfield Park
Emma
Northanger Abbey
Persuasion

Take a peek

austen_books()  #all six texts in tidy format
text book
SENSE AND SENSIBILITY Sense & Sensibility
Sense & Sensibility
by Jane Austen Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
CHAPTER 1 Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
The family of Dashwood had long been settled in Sussex. Their estate Sense & Sensibility
was large, and their residence was at Norland Park, in the centre of Sense & Sensibility
their property, where, for many generations, they had lived in so Sense & Sensibility
respectable a manner as to engage the general good opinion of their Sense & Sensibility
surrounding acquaintance. The late owner of this estate was a single Sense & Sensibility
man, who lived to a very advanced age, and who for many years of his Sense & Sensibility
life, had a constant companion and housekeeper in his sister. But her Sense & Sensibility
death, which happened ten years before his own, produced a great Sense & Sensibility
alteration in his home; for to supply her loss, he invited and received Sense & Sensibility
into his house the family of his nephew Mr. Henry Dashwood, the legal Sense & Sensibility
inheritor of the Norland estate, and the person to whom he intended to Sense & Sensibility
bequeath it. In the society of his nephew and niece, and their Sense & Sensibility
children, the old Gentleman’s days were comfortably spent. His Sense & Sensibility
attachment to them all increased. The constant attention of Mr. and Sense & Sensibility
Mrs. Henry Dashwood to his wishes, which proceeded not merely from Sense & Sensibility
interest, but from goodness of heart, gave him every degree of solid Sense & Sensibility
comfort which his age could receive; and the cheerfulness of the Sense & Sensibility
children added a relish to his existence. Sense & Sensibility
Sense & Sensibility
By a former marriage, Mr. Henry Dashwood had one son: by his present Sense & Sensibility
lady, three daughters. The son, a steady respectable young man, was Sense & Sensibility
amply provided for by the fortune of his mother, which had been large, Sense & Sensibility
and half of which devolved on him on his coming of age. By his own Sense & Sensibility
marriage, likewise, which happened soon afterwards, he added to his Sense & Sensibility
wealth. To him therefore the succession to the Norland estate was not Sense & Sensibility
so really important as to his sisters; for their fortune, independent Sense & Sensibility
of what might arise to them from their father’s inheriting that Sense & Sensibility
property, could be but small. Their mother had nothing, and their Sense & Sensibility
father only seven thousand pounds in his own disposal; for the Sense & Sensibility
remaining moiety of his first wife’s fortune was also secured to her Sense & Sensibility
child, and he had only a life-interest in it. Sense & Sensibility
Sense & Sensibility
The old gentleman died: his will was read, and like almost every other Sense & Sensibility
will, gave as much disappointment as pleasure. He was neither so Sense & Sensibility
unjust, nor so ungrateful, as to leave his estate from his nephew;–but Sense & Sensibility
he left it to him on such terms as destroyed half the value of the Sense & Sensibility
bequest. Mr. Dashwood had wished for it more for the sake of his wife Sense & Sensibility
and daughters than for himself or his son;–but to his son, and his Sense & Sensibility
son’s son, a child of four years old, it was secured, in such a way, as Sense & Sensibility
to leave to himself no power of providing for those who were most dear Sense & Sensibility
to him, and who most needed a provision by any charge on the estate, or Sense & Sensibility
by any sale of its valuable woods. The whole was tied up for the Sense & Sensibility
benefit of this child, who, in occasional visits with his father and Sense & Sensibility
mother at Norland, had so far gained on the affections of his uncle, by Sense & Sensibility
such attractions as are by no means unusual in children of two or three Sense & Sensibility
years old; an imperfect articulation, an earnest desire of having his Sense & Sensibility
own way, many cunning tricks, and a great deal of noise, as to outweigh Sense & Sensibility
all the value of all the attention which, for years, he had received Sense & Sensibility
from his niece and her daughters. He meant not to be unkind, however, Sense & Sensibility
and, as a mark of his affection for the three girls, he left them a Sense & Sensibility
thousand pounds a-piece. Sense & Sensibility
Sense & Sensibility
Mr. Dashwood’s disappointment was, at first, severe; but his temper was Sense & Sensibility
cheerful and sanguine; and he might reasonably hope to live many years, Sense & Sensibility
and by living economically, lay by a considerable sum from the produce Sense & Sensibility
of an estate already large, and capable of almost immediate Sense & Sensibility
improvement. But the fortune, which had been so tardy in coming, was Sense & Sensibility
his only one twelvemonth. He survived his uncle no longer; and ten Sense & Sensibility
thousand pounds, including the late legacies, was all that remained for Sense & Sensibility
his widow and daughters. Sense & Sensibility
Sense & Sensibility
His son was sent for as soon as his danger was known, and to him Mr.  Sense & Sensibility
Dashwood recommended, with all the strength and urgency which illness Sense & Sensibility
could command, the interest of his mother-in-law and sisters. Sense & Sensibility
Sense & Sensibility
Mr. John Dashwood had not the strong feelings of the rest of the Sense & Sensibility
family; but he was affected by a recommendation of such a nature at Sense & Sensibility
such a time, and he promised to do every thing in his power to make Sense & Sensibility
them comfortable. His father was rendered easy by such an assurance, Sense & Sensibility
and Mr. John Dashwood had then leisure to consider how much there might Sense & Sensibility
prudently be in his power to do for them. Sense & Sensibility
Sense & Sensibility
He was not an ill-disposed young man, unless to be rather cold hearted Sense & Sensibility
and rather selfish is to be ill-disposed: but he was, in general, well Sense & Sensibility
respected; for he conducted himself with propriety in the discharge of Sense & Sensibility
his ordinary duties. Had he married a more amiable woman, he might Sense & Sensibility
have been made still more respectable than he was:–he might even have Sense & Sensibility
been made amiable himself; for he was very young when he married, and Sense & Sensibility
very fond of his wife. But Mrs. John Dashwood was a strong caricature Sense & Sensibility
of himself;–more narrow-minded and selfish. Sense & Sensibility
Sense & Sensibility
When he gave his promise to his father, he meditated within himself to Sense & Sensibility
increase the fortunes of his sisters by the present of a thousand Sense & Sensibility
pounds a-piece. He then really thought himself equal to it. The Sense & Sensibility
prospect of four thousand a-year, in addition to his present income, Sense & Sensibility
besides the remaining half of his own mother’s fortune, warmed his Sense & Sensibility
heart, and made him feel capable of generosity.– “Yes, he would give Sense & Sensibility
them three thousand pounds: it would be liberal and handsome! It would Sense & Sensibility
be enough to make them completely easy. Three thousand pounds! he Sense & Sensibility
could spare so considerable a sum with little inconvenience.”– He Sense & Sensibility
thought of it all day long, and for many days successively, and he did Sense & Sensibility
not repent. Sense & Sensibility
Sense & Sensibility
No sooner was his father’s funeral over, than Mrs. John Dashwood, Sense & Sensibility
without sending any notice of her intention to her mother-in-law, Sense & Sensibility
arrived with her child and their attendants. No one could dispute her Sense & Sensibility
right to come; the house was her husband’s from the moment of his Sense & Sensibility
father’s decease; but the indelicacy of her conduct was so much the Sense & Sensibility
greater, and to a woman in Mrs. Dashwood’s situation, with only common Sense & Sensibility
feelings, must have been highly unpleasing;–but in HER mind there was Sense & Sensibility
a sense of honor so keen, a generosity so romantic, that any offence of Sense & Sensibility
the kind, by whomsoever given or received, was to her a source of Sense & Sensibility
immovable disgust. Mrs. John Dashwood had never been a favourite with Sense & Sensibility
any of her husband’s family; but she had had no opportunity, till the Sense & Sensibility
present, of shewing them with how little attention to the comfort of Sense & Sensibility
other people she could act when occasion required it. Sense & Sensibility
Sense & Sensibility
So acutely did Mrs. Dashwood feel this ungracious behaviour, and so Sense & Sensibility
earnestly did she despise her daughter-in-law for it, that, on the Sense & Sensibility
arrival of the latter, she would have quitted the house for ever, had Sense & Sensibility
not the entreaty of her eldest girl induced her first to reflect on the Sense & Sensibility
propriety of going, and her own tender love for all her three children Sense & Sensibility
determined her afterwards to stay, and for their sakes avoid a breach Sense & Sensibility
with their brother. Sense & Sensibility
Sense & Sensibility
Elinor, this eldest daughter, whose advice was so effectual, possessed Sense & Sensibility
a strength of understanding, and coolness of judgment, which qualified Sense & Sensibility
her, though only nineteen, to be the counsellor of her mother, and Sense & Sensibility
enabled her frequently to counteract, to the advantage of them all, Sense & Sensibility
that eagerness of mind in Mrs. Dashwood which must generally have led Sense & Sensibility
to imprudence. She had an excellent heart;–her disposition was Sense & Sensibility
affectionate, and her feelings were strong; but she knew how to govern Sense & Sensibility
them: it was a knowledge which her mother had yet to learn; and which Sense & Sensibility
one of her sisters had resolved never to be taught. Sense & Sensibility
Sense & Sensibility
Marianne’s abilities were, in many respects, quite equal to Elinor’s. Sense & Sensibility
She was sensible and clever; but eager in everything: her sorrows, her Sense & Sensibility
joys, could have no moderation. She was generous, amiable, Sense & Sensibility
interesting: she was everything but prudent. The resemblance between Sense & Sensibility
her and her mother was strikingly great. Sense & Sensibility
Sense & Sensibility
Elinor saw, with concern, the excess of her sister’s sensibility; but Sense & Sensibility
by Mrs. Dashwood it was valued and cherished. They encouraged each Sense & Sensibility
other now in the violence of their affliction. The agony of grief Sense & Sensibility
which overpowered them at first, was voluntarily renewed, was sought Sense & Sensibility
for, was created again and again. They gave themselves up wholly to Sense & Sensibility
their sorrow, seeking increase of wretchedness in every reflection that Sense & Sensibility
could afford it, and resolved against ever admitting consolation in Sense & Sensibility
future. Elinor, too, was deeply afflicted; but still she could Sense & Sensibility
struggle, she could exert herself. She could consult with her brother, Sense & Sensibility
could receive her sister-in-law on her arrival, and treat her with Sense & Sensibility
proper attention; and could strive to rouse her mother to similar Sense & Sensibility
exertion, and encourage her to similar forbearance. Sense & Sensibility
Sense & Sensibility
Margaret, the other sister, was a good-humored, well-disposed girl; but Sense & Sensibility
as she had already imbibed a good deal of Marianne’s romance, without Sense & Sensibility
having much of her sense, she did not, at thirteen, bid fair to equal Sense & Sensibility
her sisters at a more advanced period of life. Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
CHAPTER 2 Sense & Sensibility
Sense & Sensibility
Sense & Sensibility
Mrs. John Dashwood now installed herself mistress of Norland; and her Sense & Sensibility
mother and sisters-in-law were degraded to the condition of visitors. Sense & Sensibility
As such, however, they were treated by her with quiet civility; and by Sense & Sensibility
her husband with as much kindness as he could feel towards anybody Sense & Sensibility
beyond himself, his wife, and their child. He really pressed them, Sense & Sensibility
with some earnestness, to consider Norland as their home; and, as no Sense & Sensibility
plan appeared so eligible to Mrs. Dashwood as remaining there till she Sense & Sensibility
could accommodate herself with a house in the neighbourhood, his Sense & Sensibility
invitation was accepted. Sense & Sensibility
Sense & Sensibility
A continuance in a place where everything reminded her of former Sense & Sensibility
delight, was exactly what suited her mind. In seasons of cheerfulness, Sense & Sensibility
no temper could be more cheerful than hers, or possess, in a greater Sense & Sensibility
degree, that sanguine expectation of happiness which is happiness Sense & Sensibility
itself. But in sorrow she must be equally carried away by her fancy, Sense & Sensibility
and as far beyond consolation as in pleasure she was beyond alloy. Sense & Sensibility
Sense & Sensibility
Mrs. John Dashwood did not at all approve of what her husband intended Sense & Sensibility
to do for his sisters. To take three thousand pounds from the fortune Sense & Sensibility
of their dear little boy would be impoverishing him to the most Sense & Sensibility
dreadful degree. She begged him to think again on the subject. How Sense & Sensibility
could he answer it to himself to rob his child, and his only child too, Sense & Sensibility
of so large a sum? And what possible claim could the Miss Dashwoods, Sense & Sensibility
who were related to him only by half blood, which she considered as no Sense & Sensibility
relationship at all, have on his generosity to so large an amount. It Sense & Sensibility
was very well known that no affection was ever supposed to exist Sense & Sensibility
between the children of any man by different marriages; and why was he Sense & Sensibility
to ruin himself, and their poor little Harry, by giving away all his Sense & Sensibility
money to his half sisters? Sense & Sensibility
Sense & Sensibility
“It was my father’s last request to me,” replied her husband, “that I Sense & Sensibility
should assist his widow and daughters.” Sense & Sensibility
Sense & Sensibility
“He did not know what he was talking of, I dare say; ten to one but he Sense & Sensibility

Emma

cat(emma[1:50], sep = "\n")
## EMMA
## 
## By Jane Austen
## 
## 
## 
## 
## VOLUME I
## 
## 
## 
## CHAPTER I
## 
## 
## Emma Woodhouse, handsome, clever, and rich, with a comfortable home
## and happy disposition, seemed to unite some of the best blessings of
## existence; and had lived nearly twenty-one years in the world with very
## little to distress or vex her.
## 
## She was the youngest of the two daughters of a most affectionate,
## indulgent father; and had, in consequence of her sister's marriage, been
## mistress of his house from a very early period. Her mother had died
## too long ago for her to have more than an indistinct remembrance of
## her caresses; and her place had been supplied by an excellent woman as
## governess, who had fallen little short of a mother in affection.
## 
## Sixteen years had Miss Taylor been in Mr. Woodhouse's family, less as a
## governess than a friend, very fond of both daughters, but particularly
## of Emma. Between _them_ it was more the intimacy of sisters. Even before
## Miss Taylor had ceased to hold the nominal office of governess, the
## mildness of her temper had hardly allowed her to impose any restraint;
## and the shadow of authority being now long passed away, they had been
## living together as friend and friend very mutually attached, and Emma
## doing just what she liked; highly esteeming Miss Taylor's judgment, but
## directed chiefly by her own.
## 
## The real evils, indeed, of Emma's situation were the power of having
## rather too much her own way, and a disposition to think a little too
## well of herself; these were the disadvantages which threatened alloy to
## her many enjoyments. The danger, however, was at present so unperceived,
## that they did not by any means rank as misfortunes with her.
## 
## Sorrow came--a gentle sorrow--but not at all in the shape of any
## disagreeable consciousness.--Miss Taylor married. It was Miss Taylor's
## loss which first brought grief. It was on the wedding-day of this
## beloved friend that Emma first sat in mournful thought of any
## continuance. The wedding over, and the bride-people gone, her father and
## herself were left to dine together, with no prospect of a third to cheer
## a long evening. Her father composed himself to sleep after dinner, as
## usual, and she had then only to sit and think of what she had lost.



Homework

First, I will import the ‘sticks_messy.txt’ from my data folder. Then, I will write a simple cleaning function, which chains together a few cleaning commands (such as removing punctuation or various stop words). Then, I will split the cleaned version into one word per row. Next, I will determine how many tokens are in the file before and after cleaning (using ” ” as a separator). Finally, I will determine the TTR.

Import “sticks_messy.txt” from my data folder

sticks_messy <- readLines("data/sticks_messy.txt")
print(sticks_messy)
## [1] "Every 98 year 89769hlj Thanksgiving n@! ight O we flocked OIHJOIJ out behind : %^Dad as he KPOANL dragged the Santa suit to the road and draped it over a kind of crucifix he'd built out of metal pole in the yard. Super 352 %% Bowl week      the pole was dressed in a jersey and Rod's helmet and << > Rod had to clear it with Dad if he wanted ;! to take the helmet off. On the Fourth of July the pole was Uncle Sam, on Veteran’s Day a soldier,  on Halloween a ghost. The pole was Dad's only concession to       glee. We were allowed a single !)()_. Crayola from the box at a (& time. One Christmas Eve he shrieked at Kimmie for #@#%  wasting an apple slice. He hovered over us as we poured ketchup saying: good enough good enough good enough. Birthday parties ()(@ consisted of cupcakes, no ice cream. The first time I brought a date over she said: what's with Your dad and that pole? and I sat there blinking."                                                                                                                                                                                                                                                                                                                                                                                                                      
## [2] "We left h! +m++ mome, married,  had children of our own, found the seeds of meanness blooming also within us. Dad began dressing the pole with more >> mcomplexity and less discernible logic. He draped some kind of fur over it on Groundhog Day and lugged out a floodlight to ensure a shadow. When an earthquake struck Chile he lay the pole on its side and spray painted a rift in ++ the earth. Mom died and he dressed == _ the pole as Death and hung from the crossbar photos of Mom as a          baby. We'd stop by and find odd , M :? talismans from his youth arranged around the base: army medals, 17 theater tickets, old sweatshirts, tubes of Mom's makeup. One autumn he painted the pole bright yellow. He covered it with cotton swabs that winter for warmth and provided offspring by hammering in six crossed sticks around the yard. He ran lengths of string     between the pole and the sticks, and taped to the string letters of apology, admissions of error, pleas for understanding, all written in $  a frantic hand on index cards. He painted a sign saying < }{} LOVE and hung it from the pole and another that said FORGIVE? and then he died in the hall with the radio on and we  ?>? sold the house to a young couple who yanked out +++m | ? the pole and the sticks and #*1 left them by the road @ on garbage day."

Cleaning function

# create function with following cleaning: remove punctuation, remove the words
# 'a' and 'the', remove excess spaces, remove numbers
cleaning_func <- function(text) {
    x <- text
    x <- gsub("[[:punct:]]", " ", x)  #removes punctuation
    x <- gsub("\\ba\\b", " ", x)  #removes the word 'a'
    x <- gsub("\\bthe\\b", " ", x)  #removes the word 'the'
    x <- gsub(" {2,}", " ", x)  #removes excess spaces
    x <- gsub("[0-9]+", " ", x)  #removes digits 0-9
    return(x)
}

# test function
cleaning_func("This      is a test of the cleaning     function2!")
## [1] "This is test of cleaning function  "
# apply cleaning function to sticks_messy
sticks_cleaned <- cleaning_func(sticks_messy)
sticks_cleaned
## [1] "Every   year  hlj Thanksgiving n ight O we flocked OIHJOIJ out behind Dad as he KPOANL dragged Santa suit to road and draped it over kind of crucifix he d built out of metal pole in yard Super   Bowl week pole was dressed in jersey and Rod s helmet and Rod had to clear it with Dad if he wanted to take helmet off On Fourth of July pole was Uncle Sam on Veteran s Day soldier on Halloween ghost The pole was Dad s only concession to glee We were allowed single Crayola from box at time One Christmas Eve he shrieked at Kimmie for wasting an apple slice He hovered over us as we poured ketchup saying good enough good enough good enough Birthday parties consisted of cupcakes no ice cream The first time I brought date over she said what s with Your dad and that pole and I sat there blinking "                                                                                                                                                                                                                                                                                                                                                                
## [2] "We left h m mome married had children of our own found seeds of meanness blooming also within us Dad began dressing pole with more mcomplexity and less discernible logic He draped some kind of fur over it on Groundhog Day and lugged out floodlight to ensure shadow When an earthquake struck Chile he lay pole on its side and spray painted rift in earth Mom died and he dressed pole as Death and hung from crossbar photos of Mom as baby We d stop by and find odd M talismans from his youth arranged around base army medals   theater tickets old sweatshirts tubes of Mom s makeup One autumn he painted pole bright yellow He covered it with cotton swabs that winter for warmth and provided offspring by hammering in six crossed sticks around yard He ran lengths of string between pole and sticks and taped to string letters of apology admissions of error pleas for understanding all written in frantic hand on index cards He painted sign saying LOVE and hung it from pole and another that said FORGIVE and then he died in hall with radio on and we sold house to young couple who yanked out m pole and sticks and   left them by road on garbage day "

Tokenize the cleaned text - one word per row

# create data frame with sticks_cleaned and split to one word per row
sticks_cleaned_df <- data.frame(word = unlist(strsplit(sticks_cleaned, "\\s+")))

sticks_cleaned_df
x
Every
year
hlj
Thanksgiving
n
ight
O
we
flocked
OIHJOIJ
out
behind
Dad
as
he
KPOANL
dragged
Santa
suit
to
road
and
draped
it
over
kind
of
crucifix
he
d
built
out
of
metal
pole
in
yard
Super
Bowl
week
pole
was
dressed
in
jersey
and
Rod
s
helmet
and
Rod
had
to
clear
it
with
Dad
if
he
wanted
to
take
helmet
off
On
Fourth
of
July
pole
was
Uncle
Sam
on
Veteran
s
Day
soldier
on
Halloween
ghost
The
pole
was
Dad
s
only
concession
to
glee
We
were
allowed
single
Crayola
from
box
at
time
One
Christmas
Eve
he
shrieked
at
Kimmie
for
wasting
an
apple
slice
He
hovered
over
us
as
we
poured
ketchup
saying
good
enough
good
enough
good
enough
Birthday
parties
consisted
of
cupcakes
no
ice
cream
The
first
time
I
brought
date
over
she
said
what
s
with
Your
dad
and
that
pole
and
I
sat
there
blinking
We
left
h
m
mome
married
had
children
of
our
own
found
seeds
of
meanness
blooming
also
within
us
Dad
began
dressing
pole
with
more
mcomplexity
and
less
discernible
logic
He
draped
some
kind
of
fur
over
it
on
Groundhog
Day
and
lugged
out
floodlight
# 367 rows

Compare list size before and after cleaning

# Determine number of rows in tokenized messy list
sticks_messy_df <- data.frame(word = unlist(strsplit(sticks_messy, "\\s+")))

sticks_messy_df
x
Every
98
year
89769hlj
Thanksgiving
n@!
ight
O
we
flocked
OIHJOIJ
out
behind
:
%^Dad
as
he
KPOANL
dragged
the
Santa
suit
to
the
road
and
draped
it
over
a
kind
of
crucifix
he’d
built
out
of
metal
pole
in
the
yard.
Super
352
%%
Bowl
week
the
pole
was
dressed
in
a
jersey
and
Rod’s
helmet
and
<<
>
Rod
had
to
clear
it
with
Dad
if
he
wanted
;!
to
take
the
helmet
off.
On
the
Fourth
of
July
the
pole
was
Uncle
Sam,
on
Veteran’s
Day
a
soldier,
on
Halloween
a
ghost.
The
pole
was
Dad’s
only
concession
to
glee.
We
were
allowed
a
single
!)()_.
Crayola
from
the
box
at
a
(&
time.
One
Christmas
Eve
he
shrieked
at
Kimmie
for
#@#%
wasting
an
apple
slice.
He
hovered
over
us
as
we
poured
ketchup
saying:
good
enough
good
enough
good
enough.
Birthday
parties
()(@
consisted
of
cupcakes,
no
ice
cream.
The
first
time
I
brought
a
date
over
she
said:
what’s
with
Your
dad
and
that
pole?
and
I
sat
there
blinking.
We
left
h!
+m++
mome,
married,
had
children
of
our
own,
found
the
seeds
of
meanness
blooming
also
within
us.
Dad
began
dressing
the
nrow(sticks_messy_df)  #counts number of rows
## [1] 427
# 427 rows

# Compare to sticks_cleaned_df
sticks_cleaned_df
x
Every
year
hlj
Thanksgiving
n
ight
O
we
flocked
OIHJOIJ
out
behind
Dad
as
he
KPOANL
dragged
Santa
suit
to
road
and
draped
it
over
kind
of
crucifix
he
d
built
out
of
metal
pole
in
yard
Super
Bowl
week
pole
was
dressed
in
jersey
and
Rod
s
helmet
and
Rod
had
to
clear
it
with
Dad
if
he
wanted
to
take
helmet
off
On
Fourth
of
July
pole
was
Uncle
Sam
on
Veteran
s
Day
soldier
on
Halloween
ghost
The
pole
was
Dad
s
only
concession
to
glee
We
were
allowed
single
Crayola
from
box
at
time
One
Christmas
Eve
he
shrieked
at
Kimmie
for
wasting
an
apple
slice
He
hovered
over
us
as
we
poured
ketchup
saying
good
enough
good
enough
good
enough
Birthday
parties
consisted
of
cupcakes
no
ice
cream
The
first
time
I
brought
date
over
she
said
what
s
with
Your
dad
and
that
pole
and
I
sat
there
blinking
We
left
h
m
mome
married
had
children
of
our
own
found
seeds
of
meanness
blooming
also
within
us
Dad
began
dressing
pole
with
more
mcomplexity
and
less
discernible
logic
He
draped
some
kind
of
fur
over
it
on
Groundhog
Day
and
lugged
out
floodlight
nrow(sticks_cleaned_df)  #counts number of rows
## [1] 367
# 367 rows

Determine TTR of cleaned data frame

# Determine number of unique words in sticks_cleaned_df
sticks_unique <- length(unique(sticks_cleaned_df$word))
sticks_unique
## [1] 224
# 224

# Calculate TTR as unique / total
sticks_TTR <- sticks_unique/nrow(sticks_cleaned_df)
sticks_TTR
## [1] 0.6103542
# 0.6103542